Application of machine learning to iterative and multimodality image reconstruction

ABSTRACT

A method for machine learning based ultrasound image reconstruction can include receiving, at a reconstruction engine, imaging data; generating an initial estimate for a transmission image via a neural network trained (machine or self-learning) on paired transmission ultrasound and reflection ultrasound data; and performing image reconstruction using the initial estimate to generate transmission ultrasound images. The image reconstruction can generate higher quality transmission ultrasound when carried out by using the initial estimate as the starting point for iterative image reconstruction and using transmission data obtained via conventional transmission ultrasound frequencies (e.g. from 0.8 MHz to 1.5 MHz).

BACKGROUND

Medical imaging involves acquisition of data and image reconstructionfrom the data. Image reconstruction from acquired data can be treated asan inverse problem that can be solved via iterative methods. Often, anapproximate solution—an assumed image or an initial estimate—is firstobtained and then a better reconstruction is generated using multipleiteration steps. The iterative steps can include calculating thedifference between the measured and the modeled fields utilizing all theacquired projections data, and updating the image based on comparison,resulting in progressive improvement of image quality and also reductionin reconstruction image artifacts. Image reconstruction may be carriedout for imaging modalities such as magnetic resonance imaging (MRI),positron emission tomography (PET), computed tomography (CT), andultrasound.

Ultrasound imaging uses high-frequency sound waves to view internalstructures such as tissues and blood flow. A transducer transmitsultrasound waves and a receiver captures the waves after reflecting fromor transmitting through the tissue. A quantitative transmissionultrasound system performs both transmission ultrasound and reflectionultrasound methods to gather data.

BRIEF SUMMARY

Machine learning based image reconstruction is described. Theapplication of machine learning to iterative and multimodality imagereconstruction can be considered “efficient” image reconstruction due tothe ability to reconstruct images with fewer data points of raw data. Inthe context of ultrasound imaging, efficient image reconstruction doesnot require the low frequency transmission ultrasound data in order toreconstruct transmission images such as speed of sound images. Rather,reflection data alone or in combination with transmission data obtainedvia more conventional ultrasound frequencies (e.g., from 0.8 MHz to 1.5MHz in one instance) can be used to generate suitable transmissionimages.

An image reconstruction system can include a communication interfacereceiving imaging data from a scanner system and a reconstructionengine. The reconstruction engine can include at least one neuralnetwork, a processor, and storage media storing instructions forperforming image reconstruction when executed by the processor. Theneural network includes a GAN-based algorithm (resulting in a “GAN-basedneural network”) that receives at least a component of the imaging datafrom the scanner system via the communication interface.

The image reconstruction system can further include a repository thatstores a training data set and/or images collected from use of animaging modality system (e.g., quantitative transmission ultrasoundsystems) over time for training or self-learning of the neural networkof the reconstruction engine.

In some cases, the system further includes a discriminator for use intraining the one or more neural networks. For example, the repositorycan store a training set of reflection and transmission images and/orraw data The training or self-learning of the neural network can beaccomplished by using the discriminator to compare the transmission data(corresponding to the reflection data) from the repository to thesynthetic transmission data generated from the corresponding reflectiondata and output a set of errors that are used to adjust weights in theGAN-based algorithm for the neural network.

In some cases, one or more application programming interfaces can beprovided as part of the image reconstruction system to support thirdparty submission of training data and support third party submission ofimage reconstruction algorithms.

A method for image reconstruction can include receiving, at areconstruction engine, imaging data; generating, using at least oneGAN-based neural network, intermediate synthetic transmission data fromat least a component of the imaging data; performing iterative methodson at least the intermediate synthetic transmission data until a desiredaccuracy is reached; and upon reaching the desired accuracy, outputtingfinal synthetic transmission data. When the imaging data includesreflection data, an initial/intermediate synthetic estimate for atransmission image can be generated using a neural network and GAN-basedalgorithm (a “GAN-based neural network”) that is trained (machine orself-learning) on imaging data (e.g., paired reflection and transmissiondata such as available via quantitative transmission ultrasound); and aniterative method for image reconstruction is performed using theinitial/intermediate synthetic estimate to generate increasinglyaccurate transmission images until a desired accuracy is reached. Insome cases, transmission data (e.g., of a conventional frequency rangesuch as starting above 1.5 MHz) can be included with the receivedimaging data and the image reconstruction can be carried out using thetransmission data and the initial/intermediate synthetic estimate. Insome cases, two types of reflection data (e.g., a reflection image(s)and a set of reflection data) can be processed by GAN-based neuralnetworks trained on the imaging data to create two types ofinitial/intermediate synthetic transmission data (e.g., a synthetictransmission image(s) and a set of synthetic transmission data). Whenonly reflection data is received, the image reconstruction can becarried out using synthetic transmission data and theinitial/intermediate synthetic estimate.

A method for training a neural network can include receiving trainingimaging data, the imaging data comprising paired transmission data andreflection data; generating, using a GAN-based neural network, synthetictransmission data from the reflection data; comparing transmission datathat is paired with the reflection data with the synthetic transmissiondata to determine a set of errors; and adjusting weights of theGAN-based neural network using the set of errors. The method fortraining a neural network can further or alternatively include receivingpaired transmission images and reflection images, generating using aGAN-based neural network, an initial guess of a synthetic transmissionimage from a reflection image; performing iterative methods using speedof sound data corresponding to a transmission image that is paired withthe reflection image and the initial guess of the synthetic transmissionimage until a desired accuracy is reached; upon reaching the desiredaccuracy, outputting a final synthetic transmission image; comparing thetransmission image that is paired with the reflection image with thefinal synthetic transmission image to determine a set of errors; andadjusting the weights of the GAN-based neural network using the set oferrors.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an operating environment for machine learning basedimage reconstruction.

FIGS. 2A and 2B illustrate example reconstruction engine configurations.

FIG. 3 illustrates an example scenario using machine learning basedimage reconstruction.

FIG. 4 illustrates an example process for training a feature set of aneural network using paired data.

DETAILED DISCLOSURE

The application of machine learning or self-learning to iterative andmultimodality image reconstruction is described. Iterative imagereconstruction refers to the methods for the image reconstruction.Multimodality refers to the ability of using different types of inputparameters for an image. For example, CT and MRI data may be used toreconstruct a MRI image; reflection data may be used to reconstruct aspeed of sound image; or a first bandwidth range of data may be used toreconstruct an image covering a larger or different bandwidth range ofdata. The resulting output that may be of different modality orcharacteristics is referred to as “synthetic” output, which may be rawdata or images. Synthetic information is information of a type thatemulates the real information of that type but is either made from adifferent type of information or is of the same type but with differentaspects (e.g., frequencies).

The ability to reconstruct images from suboptimal inputs as provided bycertain embodiments of the described systems and techniques for“efficient” image reconstruction is highly advantageous. For example,certain embodiment of the described systems and techniques can allow fordata received from cheaper, older, or more general machines to be usedto reconstruct useful and detailed images. In some cases, mobilescanners can be used, and the more complex reconstruction can be carriedout on separate systems such as described herein. The mobile scannersmay be reflection ultrasound scanners and/or transmission ultrasoundscanners with transmission frequencies above a certain threshold (e.g.,starting between 800 kHz and 1.5 MHz). Transmission ultrasound scannerswith larger frequency ranges (e.g., starting at lower frequencies suchas 300 kHz) generally produce clearer and higher quality images;however, such transmission ultrasound scanners tend to requireunconventional, finer or larger hardware to collect the transmissiondata, which often results in a more expensive machine. The describedsystems and techniques enable scanners of lower bandwidth to produceadequate images (which may be clearer or appear higher quality than whatis usually able to be reconstructed from the data), which can furthersupport availability of ultrasound images in a wide variety ofapplications and environments.

Indeed, in some cases, the described methods can be used to produce veryrough scans, from relatively limited amount of data/projections, in ashort period of time. This is especially useful as a real-time displaywhile scanning (e.g., “scout scan”) in order to aid in patientpositioning. The scan can be cancelled in short order if the patientpositioning is inadequate, and this could be determined quickly—farbefore the image would have been fully acquired and/or processed throughmore conventional means.

FIG. 1 illustrates an operating environment for machine learning basedimage reconstruction. Referring to FIG. 1, the operating environment caninclude an image reconstruction system 110. The system 110 can include acommunication interface 105 that can receive imaging data from a scannersystem 160 and a reconstruction engine 120. The system 110 can furtherinclude a storage device 140 containing training data for a neuralnetwork of reconstruction engine 120 (e.g., neural networks 122, 124),and one or more APIs 150, 152.

The system 110 can be remote, cloud-based, or locally operated.Similarly, the components of the system can exist in the same space orsome or all can be hosted in different locations (e.g., distributed). Insome cases, inputted data can be streamed in real-time from a scannersystem 160. In some cases, input data can be batch received or obtainedpost scan, for example, directly from a scanner system 160 or when thescanner system 160 transmits the input data to an intermediary system,which provides or makes available the data to the system 110 aftercompletion of a scan (or parts of the scan). Interface 105 may be anysuitable communications interface that can communicate wired orwirelessly with scanner system 160 directly or indirectly.

For ultrasound implementations, the system 110 can take in, depending onimplementation, ultrasound data as input. Ultrasound data can includetransmission data providing speed-of-sound and attenuation informationand reflection data providing reflection information. Raw data or imagesmay be received. The system 110 can, in other implementations, receiveas input various other image datatypes, such as CT or MRI. Thereconstruction engine 120 can be the part of the system that transformsthe input (e.g., any of the image data described above) into the outputof a reconstructed image. It should be understood that the system 110 iscapable of using one imaging modality datatype (e.g., MRI) and outputanother synthetic imaging modality datatype (e.g., ultrasound) with theuse of paired data, as is explained further below.

One method of machine learning based image reconstruction uses neuralnetworks. Accordingly, the reconstruction engine 120 can include atleast one neural network (such as one, both, or more than the two neuralnetworks 122, 124 illustrated in FIG. 1), a processor 126, and acomputer storage system 128 storing executable code, for example areconstruction algorithm, that can be executed by processor 126.

The one or more neural networks 122, 124 can be Generative AdversarialNetworks (GANs). Accordingly, at least one neural network includes aGAN-based algorithm that receives at least a component of the imagingdata from the communication interface 105.

The inputs to the reconstruction engine 120 as a whole can be directlyinput to the neural networks (e.g., neural networks 122, 124). In somecases, the inputs to the neural networks 122, 124 undergo datanormalization and/or conversion from 3D volumes into 2D slices. Exampleimplementations of reconstruction engines can be seen in FIGS. 2A and2B.

The neural networks 122, 124 can utilize feature sets that serve toassist in transforming one or more images and/or datasets into anotherimage and/or dataset. Features such as interfaces where thetissue/matter changes within a reflection image—for instance, skin andconnective tissue elements such as Cooper's ligaments—have a largeimpact on defining boundaries within the generated speed-of-sound images(see also U.S. application Ser. No. 15/836,576, entitled “Color CodingAn Image for Identifying Anatomy Using Quantitative TransmissionUltrasound Tomography”, for identification of features. Particularlywhen a cyclical neural network, such as a CycleGAN is used, featuressuch as interfaces between skin and water, dense and fatty tissue, cystsand solids, and physical tissue deformations in the presence of medicaltape can be useful in generating reflection images (see also U.S.application Ser. No. 15/836,576, entitled “Color Coding An Image forIdentifying Anatomy Using Quantitative Transmission UltrasoundTomography”, for identification of features).

Processor 126 can receive the output of the neural network(s) 122, 124(and additionally, in some cases, possibly a direct input to the system110, which may or may not have been used by reconstruction engine 120).Via execution of executable code, the processor 126 can perform imagereconstruction, for example, via iterative methods for imagereconstruction.

In some cases, the at least one neural network includes two neuralnetworks (e.g., neural networks 122, 124) and the imaging data receivedvia interface 105 is reflection data and a reflection image. Thedescribed neural networks can be used to estimate a transmission imagefrom a reflection image, which serves a similar purpose to low frequencytransmission data in providing a starting point for image reconstructionto be tuned by higher frequencies.

For example, one neural network 124 receives the reflection data andgenerates a set of synthetic transmission data from the reflection data.The other neural network 122 receives the reflection image and generatesan intermediate synthetic transmission image from the reflection image.The instructions for performing image reconstruction, when executed bythe processor 126, then direct the processor 126 to at least: performiterative methods on the set of synthetic transmission data and theintermediate synthetic transmission image until a desired accuracy isreached, and upon reaching the desired accuracy, output a finalsynthetic transmission image.

A similar method can also be used if both reflection and transmissiondata are available to produce a superior image by using the latest imageproduced in each iteration to perform a check on the corresponding otherimage. For example, the imaging data can be high frequency transmissiondata and a reflection image. The at least one neural network (e.g.,neural network 122) receives the reflection image and generates anintermediate synthetic transmission image from the reflection image. Theinstructions for performing image reconstruction, when executed by theprocessor 126, then direct the processor 126 to at least: performiterative methods on the high frequency transmission data and theintermediate synthetic transmission image until a desired accuracy isreached, and upon reaching the desired accuracy, output a finalsynthetic transmission image.

As can be seen from the examples illustrated above, the inputs of theprocessor 126 can be an initial guess of the image that is provided fromthe output of one of the neural network(s) 122, 124, transmission data,and/or reflection data. The output of the processor 126 can be the finalimage.

Example methods for image reconstruction can incorporate inversescattering techniques, such as but not limited to the techniquesdescribed in “3-D Nonlinear Acoustic Inverse Scattering: Algorithm andQuantitative Results” (Wiskin et al., IEEE Transactions on Ultrasonics,Ferroelectrics, and Frequency Control, Vol. 64, No. 8, August 2017).

In some cases, image generation/reconstruction can use the neuralnetwork(s) to selectively remove and/or replace relatively higher noise(or low signal to noise ratio) image data at certain frequencies and/orviewing angles.

In some cases, the neural network(s) can be tuned to selectively improveand/or enhance the image quality of a certain tissue type or a region ofinterest. For example, use of a neural-network can selectively enhancethe visibility of region with microcalcifications. In some of thesecases, image quality in the rest of the breast tissue may become worse.In these cases, the enhanced visibility in the region with themicrocalcifications may be merged with the original image (minus thecorresponding region with the microcalcifications in the original image)to create an enhanced image. However, in some cases, the image qualityin the rest of the breast tissue does not become worse and thereforedoes not need to be merged with the original image.

In some cases, system 110 can include an application programminginterface (API) 150 enabling third parties to provide reconstructionalgorithms (e.g., in the form of executable code) used by thereconstruction engine (e.g., processor 126). This API 150 can beconsidered an image reconstruction selector API. In some cases, theimage reconstruction selector API can be used to receive an imagereconstruction algorithm for execution by the processor. In some cases,the image reconstruction selector API can be used to select an existingavailable image reconstruction algorithm for execution by the processor.In some cases, the image reconstruction selector API supportsextensibility of the reconstruction engine for use in imagereconstruction of the data/images generated by the neural networks byother parties.

The storage device 140 can store training data used to generate weightsfor the neural network(s) 122, 124 of the reconstruction engine 120. Thetraining data can be the sources of the weighting for the features(e.g., interfaces such as skin and water, dense and fatty tissue, cystsand solids, and physical tissue deformations in the presence of medicaltape, and/or skin and connective tissue elements such as Cooper'sligaments) used by the neural networks in the system.

A variety of imaging modalities may be used as input data to teach theGAN-based algorithm to generate speed or reflection images. Suchmodalities include, but are not limited to, Transmission Ultrasound,Reflection Ultrasound, CT, MRI, PET, X-ray, SPECT, X-ray Mammography orany combination of multi-modalities. The system can create final imagesof these datatypes as the outputs. It may be desirable for such inputdata (i.e., training sets) to be obtained from scanning systems capableof correlating two or more imaging modalities (e.g., provide “paireddata”). The correlation may be in terms of spatial and/or temporaldimensions between the provided paired data.

The training data itself can either be sourced externally or producedinternally. In some cases, the output of the reconstruction engine 120may be fed back to the system for self-learning. A possible methodologyfor obtaining training data from paired data is described in FIG. 4. Insome cases, system 110 includes an API 152 for submitting images for useto train the neural networks 122, 124 of the reconstruction engine 120or possibly for direct implementation of weights for the neural networks122, 124. This API 152 can be considered a training data API. Thetraining data API receives training data or weights for use in trainingthe neural network.

In addition to APIs 150, 152, other APIs may be available. An API is aninterface implemented by a program code component or hardware component(hereinafter “API-implementing component”) that allows a differentprogram code component or hardware component (hereinafter “API-callingcomponent”) to access and use one or more functions, methods,procedures, data structures, classes, and/or other services provided bythe API-implementing component. An API can define one or more parametersthat are passed between the API-calling component and theAPI-implementing component.

The API is generally a set of programming instructions and standards forenabling two or more applications to communicate with each other and iscommonly implemented over the Internet as a set of Hypertext TransferProtocol (HTTP) request messages and a specified format or structure forresponse messages according to a REST (Representational state transfer)or SOAP (Simple Object Access Protocol) architecture.

The data used in the system 110 can be collected via a device such asscanner system 160, which can also be considered an optional part of thesystem. The scanner system 160 can be a machine that includes a scanner162 that takes ultrasound, MRI, or CT readings. If the scanner system160 is an ultrasound scanner, reflection data and, optionally,transmission data can be collected. For transmission ultrasound scannerimplementations, the scanner system 160 can take readings in at leastthe 800 kHz to 1.5 MHz range. The scanner system 160 can include acommunication module 164 for communicating with system 110 (or anintermediate system) to provide image data to the system 110. Thescanner system 160 can also optionally include a storage system forstorage of scanned data if the data is not streamed live.

FIGS. 2A and 2B illustrate example reconstruction engine configurations.FIG. 2A illustrates an implementation using only reflection data, andFIG. 2B illustrates an implementation using reflection data along withtransmission data. In both examples, the reconstruction engine canperform a method including receiving imaging data; generating, using atleast one GAN-based neural network, intermediate synthetic transmissiondata from at least a component of the imaging data; performing iterativemethods on at least the intermediate synthetic transmission data until adesired accuracy is reached; and upon reaching the desired accuracy,outputting final synthetic transmission data.

FIG. 2A illustrates a reconstruction engine 200 for an input set ofimaging data that includes only reflection data (which can be in theform of both raw reflection data and a reflection image) that generatesintermediate synthetic transmission data of both a set of synthetictransmission data and an initial guess for a transmission image andoutputs a synthetic transmission image as the final synthetictransmission data.

A reconstruction engine 200 that includes only reflection data as inputand outputs a synthetic transmission image is useful because areflection data input set represents something of a barebones setup—asystem that produces only reflection data can be cheaper, as an entiresensor array is excluded. Alternatively, a reconstruction engine 200that includes only reflection data as input and outputs a synthetictransmission image could be used for diagnostic purposes in the 10 mmregion (e.g., for imaging areas where the angle for that type of imagingmodality is difficult to support transmission through the tissue), astransmission data is difficult to collect at this scale. For example, itcan be difficult to capture information in the interface from apatient's chest cavity to the patient's breast.

The reflection image 204 can be created through traditional means (priorto receipt at the reconstruction engine 200 or by a component of thereconstruction engine that is not shown in the figure), and thetransmission data that one otherwise might not be able to obtain couldbe constructed through this process. A neural network 206 can receivethe reflection data 202 and output a set of synthetic transmission data210 for the same scanned object, while another neural network 208 canreceive the reflection image 204 as input and can output an initialguess for the transmission image 212. The processor (e.g., processor 126of FIG. 1) can receive the initial guess for the transmission image 212and the set of synthetic transmission data 210, execute the code (e.g.,in the form of a reconstruction algorithm) that performs the iterativemethods 214, and produce a final transmission image 216.

Accordingly, a reconstruction method can include generating, using afirst GAN-based neural network, an intermediate set of synthetictransmission data from the reflection data; generating, using a secondGAN-based neural network, an intermediate synthetic transmission image(the initial guess) from the reflection image; and performing iterativemethods on the intermediate set of the synthetic transmission data andthe intermediate synthetic transmission image until a desired accuracyis reached in order to output a final synthetic transmission image.

FIG. 2B illustrates a reconstruction image 230 for an input set ofimaging data that includes transmission data in addition to thereflection data (which can be in the form of either raw reflection datathat is then used to generate a reflection image through traditionalmeans or is received in the form of the reflection image) and thatgenerates intermediate synthetic transmission data of an initial guessfor a transmission image and outputs a transmission image as the finalsynthetic transmission data. The transmission data can possibly belimited to only relatively high frequencies, such as starting at 800 kHzto 1.5 MHz. This can be useful because low frequency data requires finerand often more expensive or bulkier equipment. The methodology as awhole has advantages over existing implementations, even when the lowfrequency transmission data is available. Since the reflection image canbe used to create an initial estimate for the transmission image fairlyquickly (e.g., 10 seconds, 30 seconds, 1 minute, 5 minutes; for example,an estimate may be possible within 1-2 minutes for a typical breastsize), the amount of time the system takes to reach a final estimate iscut down. To implement this methodology, one neural network 234 canreceive the reflection image 232 as input and output an initial guessfor the transmission image 238. This image can be used, along with thereceived transmission data 236, as an input for the processor (e.g.,processor 126 of FIG. 1) that executes the code that performs theiterative methods 240. The output can be the final transmission image242.

Accordingly, a reconstruction method can include generating, using oneGAN-based neural network, an intermediate synthetic transmission image(the initial guess) from the reflection image; and performing iterativemethods on the intermediate synthetic transmission image and receivedtransmission data until a desired accuracy is reached in order to outputa final synthetic transmission image.

FIG. 3 illustrates an example scenario using machine learning basedimage reconstruction. The environment described is a combat scenario,where medics can quickly and accurately provide treatment to soldiersusing ultrasound analysis with minimal equipment. A soldier might beinjured in the field, prompting a medic to use a simple ultrasounddevice 302. This device may only gather reflection data 312, astransmission data requires an extra receiver, and may only acquire highfrequency data, as low frequency data requires a more complicated andoften less portable device. This data can be collected and sent to thecloud (or to a server implementing the described recognition enginelocated nearby or remotely). A reconstruction engine 310, such asdescribed with respect to 120 of FIG. 1, 200 of FIG. 2A, or 230 of FIG.2B, can use the high-frequency reflection data 312 received directlyfrom the ultrasound device 302 or indirectly (e.g., via a computingsystem that can communicate over a network) as an input to performiterative methods 314 to generate an intermediate reflection image 316.Then, a neural network 320 generates an initial guess for thetransmission image 322 using the intermediate reflection image 316. Aseparate neural network 324 can receive the high-frequency reflectiondata 312 as an input to produce a set of synthetic transmission data326. The synthetic transmission data 326 and the initial guess for thetransmission image 322 are taken as inputs to an iterative method 328,which produces the finalized transmission image 330. An army doctor, notin the field, can receive these images on the doctor's computer 332. Thedoctor can make a recommendation (e.g., diagnosis) for treatment basedon the images and forward 334 the recommendation to the medic (e.g., viaa communication device 336), who can carry out (338) the instructions,possibly without ever leaving the field. Communication device 336 can bea phone, computer, or other device that can receive audio or visualinput.

FIG. 4 illustrates an example process for training a feature set of aneural network using paired data. The process can be implemented withinan image recognition system such as image recognition system 110described with respect to FIG. 1. In some cases, one or morediscriminators can be included for use in training the one or moreneural networks 222, 224. Turning to the example in FIG. 4, for acertain dataset, there can exist both reflection and transmission dataand images for a 3D volume position corresponding to a certain scannedobject. The set of transmission and reflection data at each individualposition in the 3D image space can be considered a set of paired data,as each (spatial and/or temporal) point in the reflection data cancorrespond to a single point in the transmission data. For transformingreflection data into synthetic transmission data, a neural network 410can take in reflection data 402 as an input. The output synthetictransmission data 412 can be compared against the input transmissiondata 404 using a discriminator 414 that outputs a set of errors 416 thatcan be used to adjust the weights of the neural network. This processcan occur iteratively, until a neural network that adequately constructsa set of synthetic data that is sufficiently close (e.g., within a rangeof error that comes to a resolution/resolves, etc.) to the correspondingpaired data. The neural network 410 can also be trained across multipledatasets (e.g., using paired data from different imaging modalities) tofind a set of weights that is most useful across all possible sets.

For training a neural network to transform a reflection image into atransmission image, a similar process can be employed. The neuralnetwork 430 to be trained can take the reflection image 406 as an inputand output an initial guess 432 of a transmission image. A set ofiterative methods 434 can receive the transmission data 404 (possiblyincluding only the high frequency data) and the initial guess 432 of thetransmission image to output a synthetic transmission image 436. Adiscriminator 438 can compare the synthetic transmission image to thereal transmission image 408 and produce a set of errors 440 that can beused to adjust the weights of the neural network. This process can occuriteratively, until a neural network that adequately constructs asynthetic transmission image that is sufficiently close to thecorresponding paired image. The neural network 430 can also be trainedacross multiple datasets to find a set of weights that is most usefulacross all possible sets.

Another method of training can involve a GAN-based algorithm calledCycleGAN. CycleGAN can be used to generate speed images from reflectionimages and vice-versa. Based on paired transmission and reflectionimages, CycleGAN can learn how to convert a speed image into areflection image, and a reflection image into a speed image, at the sametime. The algorithm can employ a concept where a discriminator functionis used to minimize the error in the conversion from speed, toreflection, and back to speed again. The source of this error can bedetermined by the discriminator and the weights and features for theCycleGAN can be modified to reduce the error in future iterations.Further, at the same time, an attempt can be made to minimize the errorin the conversion from reflection, to speed, and back to reflectionagain.

The training of the neural network can occur in two dimensions or threedimensions. Training in three dimensions can allow for the neuralnetwork to better differentiate between tissues as well as learn abouttissue connectivity. For instance, adjacent voxels with similar valuescan be searched in three dimensions to more easily distinguish betweentissues. This approach can improve differentiation between anatomy, forexample, between ligaments and other elements that may have similarvalues, but not similar shapes.

Using paired training leaves open the possibility for a very strong andflexible system. If two systems have data that can be paired and arerelated at all, it is possible to train the neural network transform oneinto another. If the paired data exists at runtime—in other words, ifthe system is presented with paired data such as ultrasound and MRI—thetwo can be used together to refine the model more quickly and accuratelythan if the two were to be run by neural networks without paired data,as you create two models for testing in a single cycle.

The use of paired data can also be useful when using exclusivelyultrasound data. If both reflection and transmission data is present,the system can compare the results of each iteration in the iterativemethods to the results of the other iteration. The comparison can leadto improving the speed of the formation of the final image by comparingthe two and using the commonalities as a stronger basis for the initialimage used for the next iteration. Accordingly, a method for training aneural network can include receiving training imaging data, the imagingdata comprising paired transmission data and reflection data;generating, using a GAN-based neural network, synthetic transmissiondata from the reflection data; comparing transmission data that ispaired with the reflection data with the synthetic transmission data todetermine a set of errors; and adjusting weights of the GAN-based neuralnetwork using the set of errors.

In some cases, in addition to or as an alternative, the method caninclude receiving paired transmission images and reflection images,generating using a GAN-based neural network, an initial guess of asynthetic transmission image from a reflection image; performingiterative methods using speed of sound data corresponding to atransmission image that is paired with the reflection image and theinitial guess of the synthetic transmission image until a desiredaccuracy is reached; upon reaching the desired accuracy, outputting afinal synthetic transmission image; comparing the transmission imagethat is paired with the reflection image with the final synthetictransmission image to determine a set of errors; and adjusting theweights of the GAN-based neural network using the set of errors. Initialweights for the neural networks can be received via a training dataapplication programming interface such as described with respect to API152 of FIG. 1.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims and other equivalent features and acts are intended to be withinthe scope of the claims.

1. (canceled)
 2. A system comprising: a communication interfacereceiving imaging data from a scanner system; and a reconstructionengine comprising: at least one neural network that includes a GAN-basedalgorithm that receives at least a component of the imaging data fromthe communication interface; a processor; and storage media storinginstructions for performing image reconstruction using an output of theat least one neural network when executed by the processor, wherein theat least one neural network comprises two neural networks; wherein theimaging data is reflection data and a reflection image; wherein one ofthe two neural networks receives the reflection data and generates a setof synthetic transmission data from the reflection data; and wherein theother one of the two neural networks receives the reflection image andgenerates an intermediate synthetic transmission image from thereflection image.
 3. The system of claim 2, wherein the instructions forperforming image reconstruction, when executed by the processor, directthe processor to at least: perform iterative methods on the set ofsynthetic transmission data and the intermediate synthetic transmissionimage until a desired accuracy is reached; and upon reaching the desiredaccuracy, output a final synthetic transmission image.
 4. A systemcomprising: a communication interface receiving imaging data from ascanner system; and a reconstruction engine comprising: at least oneneural network that includes a GAN-based algorithm that receives atleast a component of the imaging data from the communication interface;a processor; and storage media storing instructions for performing imagereconstruction using an output of the at least one neural network whenexecuted by the processor, wherein the imaging data is high frequencytransmission data and a reflection image; and wherein the at least oneneural network receives the reflection image and generates anintermediate synthetic transmission image from the reflection image. 5.The system of claim 4, wherein the instructions for performing imagereconstruction, when executed by the processor, direct the processor toat least: perform iterative methods on the high frequency transmissiondata and the intermediate synthetic transmission image until a desiredaccuracy is reached; and upon reaching the desired accuracy, output afinal synthetic transmission image.
 6. The system of claim 4, furthercomprising an image reconstruction selector application programminginterface (API), wherein the image reconstruction selector API receivesan image reconstruction algorithm for execution by the processor.
 7. Thesystem of claim 4, further comprising a training data applicationprogramming interface (API), wherein the training data API receivestraining data or weights for use in training the neural network.
 8. Asystem comprising: a communication interface receiving imaging data froma scanner system; and a reconstruction engine comprising: at least oneneural network that includes a GAN-based algorithm that receives atleast a component of the imaging data from the communication interface;a processor; and storage media storing instructions for performing imagereconstruction using an output of the at least one neural network whenexecuted by the processor, wherein the reconstruction engine furthercomprises a discriminator and a repository of corresponding reflectionand transmission data; wherein a neural network of the at least oneneural network receives reflection data and generates synthetictransmission data; and wherein the discriminator compares transmissiondata corresponding to the reflection data to the synthetic transmissiondata and outputs a set of errors that are used to adjust weights in theGAN-based algorithm for the neural network.
 9. A system comprising: acommunication interface receiving imaging data from a scanner system;and a reconstruction engine comprising: at least one neural network thatincludes a GAN-based algorithm that receives at least a component of theimaging data from the communication interface; a processor; and storagemedia storing instructions for performing image reconstruction using anoutput of the at least one neural network when executed by theprocessor, further comprising a discriminator and a repository ofcorresponding reflection and transmission images; wherein a neuralnetwork of the at least one neural network receives a reflection imageand generates a synthetic transmission image; and wherein thediscriminator compares a transmission image corresponding to thereflection image to the synthetic transmission image and outputs a setof errors that are used to adjust weights in the GAN-based algorithm.10-20. (canceled)
 21. The system of claim 2, further comprising an imagereconstruction selector application programming interface (API), whereinthe image reconstruction selector API receives an image reconstructionalgorithm for execution by the processor.
 22. The system of claim 2,further comprising a training data application programming interface(API), wherein the training data API receives training data or weightsfor use in training the neural network.
 23. The system of claim 8,further comprising an image reconstruction selector applicationprogramming interface (API), wherein the image reconstruction selectorAPI receives an image reconstruction algorithm for execution by theprocessor.
 24. The system of claim 8, further comprising a training dataapplication programming interface (API), wherein the training data APIreceives training data or weights for use in training the neuralnetwork.